Enhancing a video of an event at a remote location using data acquired at the event

ABSTRACT

A system is disclosed that can be used to enhance a video of an event. Sensors are used at the event to acquire information. For example, the system can include pan, tilt and zoom sensors to acquire camera view information. This information can be added to the video signal from a camera (e.g. in the vertical blanking interval) or otherwise transmitted to a central studio. At the studio, the sensor information is used to enhance the video for broadcast. Example enhancements include drawing lines or other shapes in the video, adding advertisements to the video or adding other graphics to the video.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is directed to a system for enhancing a video ofan event using data acquired at the event.

2. Description of the Related Art

The remarkable, often astonishing, physical skills and feats of greatathletes draw millions of people every day to follow sports that rangefrom the power of American football to the grace of figure skating, fromthe speed of ice hockey to the precision of golf. Sports fans arecaptivated by the abilities of basketball players to soar to therafters, a baseball batter to drive the ball out of the park, a runnerto explode down the track, a skier to race down the hill, a running backto break through the pack and make the first down, etc. In televisingthese events, broadcasters have developed a wide repertoire oftechnologies—ranging from slow-motion replay to lipstick-sized camerasmounted on helmets—to highlight for viewers these exciting events. Otherexamples of technologies used by the broadcasters include highlightingmoving objects, showing locations on a playing field that are ofinterest to the viewer (e.g. a first down line), displaying newstatistics and adding virtual advertisements to the video.

Many of the technologies employed by broadcasters to add variousgraphics and other information to the television presentation of theevent use large amounts of equipment at the event. In some cases, atruck is needed to transport the equipment and to serve as a productioncenter for the graphical effect. Typically, such trucks includeexpensive video equipment and computers. After the event is over, all ofthe equipment is loaded on the truck and the truck is transported to thenext event. While the above-described arrangement has been successful inproviding various graphical effects, such an arrangement has itsdrawbacks because the equipment is very expensive and the systems do notmake cost efficient use of the equipment.

For example, a full set of equipment is needed for every single eventthat is televised on the same day. Thus, if a broadcaster was televisingfour football games on the same day (but, possibly, at different timesor in different regions), four sets of equipment would be needed.Purchasing four sets of equipment can be very expensive for abroadcaster and the trucks to house/transport the equipment areexpensive. Additionally, a broadcaster may not have four sets ofequipment available.

Furthermore, events tend to be located at various locations throughout ageographic region. After a television presentation of a game iscompleted, the truck must be transported to the next location. Duringtransport, the equipment is not available for use by the broadcaster.Additionally, repeatedly transporting the equipment adds to themaintenance costs of the equipment.

In sum, broadcasters spend large amounts of money for equipment that isnot available at all times and has high maintenance costs. Therefore, asystem is needed that can be used to enhance the video presentation ofan event and that makes more efficient use of the equipment necessaryfor the enhancements.

SUMMARY OF THE INVENTION

The present invention, roughly described, provides for a system toenhance a video or audio presentation of an event using data acquired atthe event. Rather than locate all of the necessary equipment at theevent, the system is broken up into at least two groups of equipment.The first group of equipment is located at the event. This first groupof equipment measures the data needed for the enhancement. The secondgroup of equipment is located at a studio location remote from theevent. The second group of equipment is used to create the actualenhancement. The enhancement can be made to video or audio. In onealternative, the first group of equipment is designed to be a small setof equipment so that it could be easily transported from event to event.

In one embodiment, the present invention includes one or more datasensors located at the event, means for transmitting data from the datasensors to a location remote from the event, and one or more processorsat the location remote from event. In one alternative, the one or moreprocessors are programmed to perform the steps of determining a positionof a target in the video and enhancing the video based on the step ofdetermining. Various data sensors can be used. In one example, the datasensor is a camera view sensor which senses information about the viewof a camera. Examples of a camera view sensor include a pan sensor, atilt sensor, a zoom sensor, a roll sensor, a focus sensor and/or a 2Xextender sensor.

In some alternatives, there may be many cameras at an event. Each cameracould include its own camera view sensor(s). The data from the variouscamera view sensors may be communicated to a data concentrator. In oneembodiment, the data concentrator can also receive an indication ofwhich camera is tallied. The system can include a tally detector whichautomatically determines which camera is tallied. The information fromthe data concentrator and the program video are transmitted to theremote location. In one embodiment, the data from the data concentratoris added to the vertical blanking interval of the program video.Alternatives, include adding the data to a digital video signal, ortransmitting the data via telephone line, cable modem, satellite,dedicated communication line, etc.

In one embodiment, the invention includes the step of sensing positiondata during the event. The position data is transmitted from the eventto the location remote from the event. The position of the target areain the video is determined using the position data. The video isenhanced based on the step of determining a position in the video. Ifthe data sensors are camera view sensors, then the position data wouldinclude camera view data. The data can also include a tally indication,a time code and/or registration information. In one alternative, thesystem can match time codes with the camera view data so that processingat the remote location can be performed by matching the appropriatecamera view data to the appropriate frame or field of video using timecodes. For some embodiments, the video includes interlaced fields suchthat an odd field and an even field comprise a frame. Other videoformats (e.g. digital video) can be used with the present invention.

In another embodiment, the present invention includes the steps ofselecting a target area in a first video image of the event, sensingcamera view data for a plurality of cameras during the event andreceiving time codes. The time codes are associated with the appropriatecamera view data. The time codes and appropriate camera view data aretransmitted to a location remote from the event. The system alsoreceives a live video of the event. The live video includes an image ofthe target area. The system determines the target area's position in thelive video using at least one of the time codes and the camera view datacorresponding to the time code. The system enhances the live video basedon the target area's position in the live video.

These and other objects and advantages of the invention will appear moreclearly from the following detailed description in which the preferredembodiment of the invention has been set forth in conjunction with thedrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of one configuration of the present invention.

FIGS. 2 and 3 are more detailed block diagrams of a second configurationof the present invention.

FIG. 4 is a flow chart describing the operation of one embodiment of thepresent invention.

FIGS. 5-7 are flow charts describing the operation of variousalternative embodiments of the present invention.

DETAILED DESCRIPTION

The present invention can be used to enhance a video presentation of asporting event or a nonsporting event. One exemplar enhancement is theblending of a graphic with a video. Video means an analog or digitalsignal depicting (or used to produce) moving images. Blending meanscombining at least a first image or video with at least second image orvideo such that the result includes all or part of the first image orvideo and all or part of the second image or video. One example of howimages are blended includes using a keyer to key one image over another.Examples of graphics that can be blended to the video presentation of anevent include adding a first down line or other marker to a video of afootball game, adding handwritten annotations to the video presentationof a sporting event, or adding logos to a video presentation of anevent. A logo can be text, graphics, or both. Examples of logos includeadvertisements, an information box, a team emblem, or any other suitablegraphic. A logo can be placed on an image of a playing field, a stadium,a stage, a player, an object of play (e.g. a ball or puck), a basketballbackboard, between the field goal uprights, in the stands, on the fans,etc.

In some embodiments, the blending of a graphic must account forocclusions. That is, if a person steps on top of or in front of an areawhere the graphic is added, the graphic should not be drawn on top ofthe person. Rather, the person should appear to be stepping on orstanding in front of the graphic.

There are many examples of systems known in the art for enhancing thevideo of a live event. U.S. patent application Ser. No. 09/160,534,filed Sep. 24, 1998, System For Enhancing a Video Presentation of a LiveEvent, incorporated herein by reference, discloses a system for adding afirst down line to a video presentation of a football game or othersporting event, and adding advertisements or other logos to a videopresentation of a live event. U.S. patent application Ser. No.08/585,145, filed Jan. 10, 1996, A System For Enhancing the TelevisionPresentation of an Object at a Sporting Event, incorporated herein byreference, discloses a system that determines the position of anobject's image in a video and enhances that object's image. U.S. Pat.No. 5,862,517 discloses a related system and is incorporated herein byreference. U.S. patent application Ser. No. 09/041,238, filed Mar. 11,1998, System For Determining The Position Of An Object, incorporatedherein by reference, discloses a system for determining the verticalposition of an object at an event and adds a graphic to the videoindicating that vertical position. U.S. patent application Ser. No.08/735,020, A Method And Apparatus For Enhancing The Broadcast of a LiveEvent, filed Oct. 22, 1996, incorporated herein by reference, disclosesa system for adding lines, advertisements and other graphics to video.Additionally, other systems are known in the art that add graphics tolive presentations of sporting events, including virtual advertisementsand other graphics.

All of the systems noted in the above paragraph, and other systems notlisted, can be used to add graphics to an existing video. Typical ofthese systems is that all of the components necessary to perform theinsertion or blending of the graphic into the video are located at theevent (e.g. at the stadium). The present invention improves on theseprior art systems by dividing the system up into at least two sets ofcomponents. A minimal set of components are transported to the actualevent. A second set of components are operated at a location remote fromthe event such as a central studio which can be used with multipleevents. By stating that equipment is at the event it is meant that theequipment is located at the site that the event is taking place. Forexample, if the event is a football game the equipment can be located onthe field, in the stadium, in the parking lot, etc. The point is thatthe equipment is local to the event. On the other hand, being remotefrom the event can mean a different part of town, in a different city,different state, etc. A production truck in the parking lot of afootball stadium is said to be at the event while a studio in anotherpart of the state is remote from the event.

FIG. 1 is a block diagram of one embodiment of the present invention.The system of FIG. 1 shows three sensors: sensor 12, sensor 14 andsensor 16. Three sensors are depicted for example purposes only. It isalso possible to practice the invention with only one sensor, twosensors or more than three sensors. The sensors are located at the eventand gather data which is used to enhance the video. In one embodimentthe sensors are used to acquire position data, which is data related tothe position or orientation of a camera and/or other object at theevent. In one alternative, the sensors can be camera view sensors. Inother alternatives, the sensors could be infrared sensors, videosensors, motion sensors, CCD's, audio sensors, light sensors or othersensors that gather data and are suitable for the particularapplication.

As stated above, one example of an appropriate sensor is a camera viewsensor. A camera view sensor is a sensor that detects data about thecamera view of a particular camera. Examples of camera view sensorsinclude a zoom sensor, a pan sensor, a tilt sensor, roll sensors,inclinometer, angle rate sensor, a 2X extender sensor, lens focussensor, or any combination of the above. In one embodiment, a zoomsensor may be a voltage output from the camera lens which indicates thestate of the zoom lens. Similarly, a 2X extender may also include anoutput which indicates which mode the extender is in and a lens mayinclude an output indicating the state of its focus. Typically,broadcast cameras are mounted upon a pan/tilt head that enables thecamera to pan and tilt. Attached to the pan and tilt head can be a pansensor and/or a tilt sensor. In one embodiment, the pan/tilt head ispart of the camera. In another embodiment, the pan/tilt head is separatefrom the camera. One embodiment uses separate pan and tilt heads.

In one embodiment, the pan sensor and the tilt sensor are opticalencoders that output a signal measured as a number of counts (orpulses), indicating the rotation of a shaft. Forty thousand (40,000)counts may represent a full 360° rotation. Thus, a processor can dividethe number of measured counts by 40,000 and multiply by 360 to determinethe pan or tilt angle in degrees. The pan and tilt sensors usetechnology known in the art and can be replaced or augmented by othersuitable pan and tilt sensors known by those skilled in the relevantart. In some embodiments, a local computer will be used with each set ofsensors. Analog to digital converters can be used to convert analogsignals from the sensors to digital signals to be used by the localcomputers.

The output of sensors 12, 14 and 16 are communicated to dataconcentrator 18. In one embodiment, a data concentrator is a device thataccepts data from multiple devices, formats the data and sends the datato another device. In one alternative, data concentrator 18 can be acomputer which accepts data from multiple sensors and combines them intoone or more signals for transmission to other computers or otherdevices.

Data concentrator 18 also receives a signal from time code generator 20.Because there may be different delays due to processing the data orblending a graphic, and because the data may be stored for future postprocessing, it is necessary in some embodiments to associate time codeswith the data and the video. In some embodiments, time codes are addedto the video before data concentrator 18. Time code generator 20 sendsthe time codes to data concentrator 18. Data concentrator 18 wouldassociate the time codes with data from the sensors. In alternativeembodiments, the time codes can be associated with the appropriate databy other devices. The time code is also associated with the video. Inone embodiment, the time code is written into the video signal for eachframe (or field) of a video. One method for adding the time code to thevideo is to add it to the vertical blanking interval of the videosignal.

If the sensors are detecting data from multiple cameras, it may beimportant to know which camera is being used for broadcast so theappropriate data can be used to do the enhancements. In one embodiment,a production truck associated with the broadcaster of an event willproduce a signal indicating which camera has been tallied for broadcast.In another embodiment, each broadcast camera will have a tally switchwhich can be monitored. In yet another embodiment, the system of FIG. 1can include a tally detector 22 which automatically determines which ofa set of cameras are tallied for broadcast. Tally detector 22 would sendan indication of which camera is tallied to data concentrator 18.

In the arrangement shown in FIG. 1, sensor 12, sensor 14, sensor 16,data concentrator 18, time code generator 20 and tally detector 22 arelocated at the event. For example, the sensors may be located at variouscamera locations in a stadium. Data concentrator 18, time code generator20 and tally detector 22 may be located at a production truck in theparking lot of the stadium.

Data concentrator 18 sends its output to a studio that is remote fromthe event via data transmission 30. Data transmission 30 can be one ofany number of alternatives for transmitting data from the event to thestudio. One example is to have modems at the event and modems at thestudio, and to transfer the data over one or more standard telephonelines. The data could also be transferred over ISDN lines, DSL lines, T1lines, cable modems or other high speed communication lines.Additionally, a dedicated WAN can be used, a fiber optic connection canbe used or a satellite can be used. Another alternative is to combinethe data from data concentrator 18 with the program video signal. If theprogram video signal is a traditional NTSC signal, then the data fromdata concentrator 18 can be added to the vertical blanking interval ofthe video signal. The data can also be added to the video signal itself.Other examples of adding the data to the video signal include encodingthe data into an audio subcarrier or encoding the data into the videosignal during horizontal retrace. As new video data formats areintroduced, the data could be added to those new video formats asappropriate. The data could also be transferred via an RF signal, amicrowave signal or any other means for communicating that are suitable.

The studio will typically include a video processing system 36, arecording system 40 and a broadcast (or distribution) system 42. Thevideo processing system 36 can include one or more processors, and othervideo equipment, that is used to enhance the video based on the datareceived from the sensors. Recording system 40 can be used to record thevideo after or before enhancement. Recording system 40 can include videotapes, DVD, laser disk, or both and other suitable media for recording.Broadcast system 42 is used to broadcast or distribute the enhancedvideo over traditional airwaves, cable tv, satellite transmission,closed circuit television, etc.

The present invention separates the video processing equipment (locatedat the studio) from the data gathering equipment (located at the event).Any of the systems described above (including those incorporated byreference) can be adapted the configuration of FIG. 1. More detailsabout many of the individual components can be found in thoseapplications incorporated by reference.

FIGS. 2 and 3 are block diagrams of a second configuration of thepresent invention. The components of FIG. 2 are located at the event andthe components of FIG. 3 are located at the studio. In alternativeembodiments, some of the components from FIG. 2 can be located at thestudio and some of the components from FIG. 3 can be located at theevent. In various embodiments, some components may also be omitted. Thesystem of FIGS. 2 and 3 can be used to enhance a video representation ofa football game. It can also be used with other sporting events ornonsporting events. On exemplar enhancement that can be accomplishedwith the system of FIGS. 2 and 3 is to blend a graphic with the video.Suitable graphics include a first down line, advertisements highlightingcurrent, previous or future locations of balls, pucks, players, cars orother moving or still objects. In many respects, the system of FIGS. 2and 3 is an adaptation, according to the present invention, of thesystem disclosed in U.S. patent application Ser. No. 09/160,534, SystemFor Enhancing a Video Presentation of a Live Event, which isincorporated by reference. The system of FIG. 2 includes three videocameras, 102, 104 and 106. Although three cameras are disclosed, thesystem can be used with more or fewer than three cameras. For example,in one embodiment, the system only includes one camera. Each camera isassociated with a set of one or more camera view sensors. For example,camera 102 includes camera view sensor(s) 108, camera 104 includescamera view sensor(s) 112 and camera 106 includes camera view sensor(s)116.

Connected to each camera may be a 2X extender, a zoom lens and a meansfor focusing the camera. The set of camera view sensors can include oneor more of the following, a sensor for detecting pan, a sensor fordetecting tilt, a sensor for detecting a roll, a sensor for detectingthe amount of zoom, a sensor for detecting the focus and a sensor fordetecting the state of the 2X extender.

Each camera is associated with a local computer. For example, camera 102is associated with computer 110, camera 104 is associated with computer114 and camera 106 is associated with computer 118. The local computerscan be a 486 processor based machine, a Pentium processor based machine,a Macintosh platform, a dedicated micro controller or another type ofcomputer/processor. Data from each of the sets of sensors 108, 112 and116 are sent to the respective local computers 110, 114 and 118. It maybe necessary to use analog to digital converters if the data from thesensors is analog in nature. In some embodiments, the zoom sensor ismerely a wire from the zoom lens to an analog to digital converter. Eachof the computers 110, 112 and 118 communicate the data from the sensorsto data concentrator 122. In one embodiment, data concentrator 122 is acomputer. With any of the devices shown in FIGS. 2 and 3, it may benecessary to use converters between RS-422 and RS-232 or other formats.One embodiment includes not using a concentrator and having dedicatedlines for each sensor. The information sent by local computers 110, 114and 1 18 include pan, tilt, zoom, focus and 2X extender data measuredfor cameras 102, 104 and 106. Data concentrator 122 may also receive asignal from tally detector 136. In one embodiment, all the signalsreceived by concentrator 122 are formatted, combined into one serialsignal and sent to data inserter 140. One embodiment includes multipledata inserters, e.g. one per camera. In another alternative, the signalsfrom the local computer can be transmitted via the microphone channel ofthe video signal from the camera. In another embodiment, dataconcentrator 122 can combine the signals into a format suitable forparallel port transmission, Ethernet transmission or other communicationmeans different than the serial signal.

The video outputs of cameras 102, 104 and 106 are sent to multiviewer130. In addition, the video outputs are also sent to a production truckused to produce the video presentation of the event. The productiontruck may receive signals from many different video cameras. Theproducer chooses which video signal(s) to broadcast. The video signal(s)chosen to be broadcast (and, perhaps, graphics added to that video) iscalled the “program video.”

The program video is communicated to multiviewer 130. In one embodiment,the cameras output an analog signal. In another embodiment, the camerasoutput a digital video signal. In another embodiment, the cameras outputanalog signals which are converted to digital signals. The system canwork with analog signals or digital signals, as long as the appropriatemultiviewer is chosen. For example, a multiviewer that can acceptdigital inputs includes the video Gainsville CVX 64Q. An example of amultiviewer that can accept analog inputs includes the Panasonic WJ-420quad unit or FOR-A MB-40E. Using digital signals may improve theperformance of tally detector 136. Multiviewer 130 combines input videosignals into one signal which is sent to tally detector 136 and monitor134. A suitable multiviewer can be used with less than four or more thanfour signals. Alternatively, if the tally detector can receive more thanone input, there may not be a need for multiviewer. Monitor 134(optional) is used by an operator to monitor the video signals beingsent to tally detector 136.

Tally detector 136 determines which (if any) of the three cameras 102,104 or 106 is tallied. A camera is said to be tallied if it is theprimary source of the video chosen by the producer to be broadcast. Withrespect to the system of FIG. 1, a camera is tallied if it is theprimary source of the video being sent as the program video. Tallydetector 136 sends to data concentrator 122 an indication of which, ifany, of the three cameras is tallied. In a system which has only onecamera, there is no need for a tally detector. In a system that has morethan three cameras, the tally detector can be designed to determinewhich of the many cameras is tallied.

The system also receives a house time code or House Sync (HouseSync/VITC) used by the broadcaster. Time code generator 124 accepts theHouse Sync/VITC signal, interprets the signal and sends the signal totime code inserter 132. Time code inserter 132 inserts time codes intothe program video and sends the program video (with time code) tomultiviewer 130, data inserter 140 and data inserter 148. The time codeinserted into the program video is used to match the camera view datawith the correct frame or field of video. In one alternative, the output(tc) of time code generator 124 is transmitted to data concentrator 122,which associates the time code with data from the sensors. One exampleof associating the time code with data is creating a data structurewhich includes a field to store the time code and add fields to storethe data.

Data concentrator 122 sends the camera view data, target data and tallyindication to data inserter 140. Data inserter 140 is a standard encoderfor inserting data into the vertical blanking interval of a videosignal. An example of a data inserter includes the TES3 Multistandard TVData Encoding Platform from Norpak Corporation of Kanata, Ontario. Datainserter 140 receives a video signal from time code inserter 132 anddata from data concentrator 122. Data inserter inserts the data fromdata concentrator 122 into the vertical blanking interval of the videosignal from time code inserter 132. In one embodiment, all of the cameraview data is transmitted on one line. In other embodiments, the cameraview data for each camera is transmitted on a separate line. The tallyindication can be transmitted on the same line as the camera view dataor on a different line.

Some systems will need to be registered. The registration process willbe discussed in more detail below. In one embodiment, data concentrator122 will store the registration information. Data concentrator 122 cancontinuously transmit the registration data to data inserter 140 suchthat the registration information is transmitted on every verticalblanking interval or once every n frames. In one embodiment, eachvertical blanking interval will include a subset of the registrationinformation.

After data inserter 140 inserts the camera view data, tally indicationand registration information into the video signal, the video signal issent to the studio via main backhaul 142. In one embodiment, datainserter 140 also inserts the time codes into the vertical blankinginterval of the video signal.

The main backhaul is the primary means for transferring the video fromthe event to the studio. In one embodiment, a dedicated fiber optic lineis used to transmit the video signal for the main backhaul. In analternative embodiment, satellite transmission can be used. In otherembodiments, other means for communicating video can be used. Mainbackhaul 142 is a standard communication channel known in the broadcasttelevision industry.

The system also includes a back up backhaul 150, which is also astandard known in the television industry. Typically, back up backhaul150 will also transmit the same video signal as main backhaul 142. Backup backhaul 150 is used in case main backhaul 142 fails. Back upbackhaul can transmit video by any of the means described for the mainbackhaul. The system of FIG. 2 also includes a second data inserter 148.Data inserter 148 can insert the camera view data, other target data,registration data, tally indication and/or time codes into the videosignal for backhaul 150. In an alternative embodiment, data inserter 148can be used to insert all or some of the video signals not tallied intoback up backhaul data stream. In one embodiment, each of the videosignals not tallied can be telemetered via the back up backhaul datastream by using known compression techniques which reduce the qualityand/or resolution of the video. In one embodiment, it is acceptable ifthe video signals being telemetered via the back up backhaul are lowerin quality because these signals will only be used for referencepurposes.

FIG. 3 shows the components of the second configuration which aretypically located at the studio. The signal from main backhaul 142 iscommunicated to recorder 178 so that the signal can be used for postprocessing after the event is over. Additionally, the signal from backup backhaul 150 is sent to recorder 184 for post-processing, or otherpurposes. The signals for main backhaul 142 and back up backhaul 150 aresent to a switch 180, which chooses one of the two signals tocommunicate to production equipment 182. Production equipment 182includes video processing equipment for adding effects to the video suchas statistics (e.g. out of town scores, time remaining), broadcasterlogos and other graphics used by the producer of the event. The effectsadded by production equipment 182 are not necessarily part of thepresent invention; however, they can be in some embodiments. Switch 180will typically select the main backhaul so that the video signal on themain backhaul will be used for broadcast. The signal from the mainbackhaul is also sent to receiver 186. The signal received from back upbackhaul 150 is sent to receiver 188. Receivers 186 and 188 are standardreceivers known in the art for reading data from the vertical blankinginterval of a video signal. Examples of receivers known in the artinclude the TTX74X PC Peripheral TV Data Broadcast Receiver and theTTX64X+External Box VBI Data Broadcast Receiver, both from NorpakCorporation. Receiver 186 and receiver 188 read the data from thevertical blanking interval and send that data to processor 190.Production equipment 182 also sends its output video signal to processor190. In one embodiment, there is no need for production equipment 182and the video signal sent from switch 180 is communicated directly toprocessor 190 and frame delay 200.

In one embodiment, the operator of the system of FIG. 3 uses processor190 to select a target area at the event to enhance. Using a model ofthe event, processor 190 determines the screen coordinates of theselected target area. Using the camera view data received from localcomputers 110, 112 and 118, processor 190 determines the position of theselected target area in the video from the camera that is tallied. Thatinformation can be used to blend a graphic with the video signal at adetermined position. Processor 190 and processor 192 work together tocreate the graphic and an alpha signal. Both the graphic and the alphasignal are sent to keyer 194.

An alpha signal that is sent to keyer 194 is also called a key signal.Each pixel in a field or frame of video may have its own key or alphavalue. The graphic signal can be sent as a YUV signal, RGB signal orYCbCr signal or other appropriate signal according to the specificationsof the keyer. Keyer 194 also receives a video signal from frame delay200. Frame delay 200 receives a video signal from either productionequipment 182 or directly from switch 180 (not shown in FIG. 3). Framedelay 200 delays the video signal to account for the processing time ofprocessors 190 and 192. In one alternative, a computer can be used toblend the graphic instead of using a keyer. For example, eitherprocessor 190 or 192 can be used, or an additional computer can be used.In some embodiments, the functions of processors 190 or 192 can becombined into one computer.

The graphics sent from processor 192 to keyer 194 is called foregroundand the signal from frame delay 200 to keyer 194 is called background.Based on the level of the alpha or key from processor 192, keyer 194determines how much foreground and background to blend on a pixel bypixel basis. Keyer 194 can blend from 100% foreground and 0% background,to 0% foreground and 100% background. In one embodiment, the key oralpha for a pixel can range from 0%-100% (or 0-1 or another range as perthe specification of the keyer). The output of keyer 194 is sent tobroadcast equipment 202 for broadcast. The output of keyer 194 is alsosent to a monitor 206 for viewing by the operator of the system.Additionally, the output of keyer 194 can be recorded.

Kill switch/watch dog timer 208, which is in communication withprocessor 192 and keyer 194, can be used by an operator to enable ordisable the keying of the graphic. In one embodiment, processor 192sends a pulse for each frame, field or other interval. The watch dogtimer will disable the keying if it does not receive the pulses after apredetermined amount of time, frames or fields.

In one embodiment, processor 190 and tally detector 136 are O2workstations from Silicon Graphics and processor 192 is an Indigo 2Impact from Silicon Graphics. In other embodiments, other suitablecomputers can be used. It is noted that these computers typicallyinclude processors, memory, disk drives, monitors, input devices, outputdevices, network interfaces, etc. In one embodiment, an Ethernet can beset up between processor 190 and 192. The Ethernet is used formaintenance purposes and communication between the processors.

FIG. 4 is a flow chart which describes the operation of one embodimentin the present invention. In step 300, the system acquires data. Lookingat the embodiment of FIG. 1, step 300 could include sensors 12, 14 and16 acquiring appropriate data. Looking at FIG. 2, step 300 could includegroups of sensors 108, 112 and 116 acquiring camera view data. In step302, the system captures video. For example, cameras 102, 104 and 106 ofFIG. 2 can be used to capture video of the event. Steps 300 and 302 canbe performed in parallel. In step 304, the system transmits relevantinformation from the event to the studio. The transmitted informationincludes the data acquired in step 300 and the video captured in step302. In step 306, the video transmitted to the studio is enhanced usingthe data acquired in step 300. Steps 300 and 302 are typically performedat the event and step 306 is performed at the studio, which is remotefrom the event. In other embodiments, steps 300 and 302 can be performedin alternative locations as technology allows.

FIG. 5 is a flow chart explaining the operation of one embodiment of thepresent invention. In step 350, the system is set up for operation.Setting up the system may include defining information needed for theoperation during the event. For example, if the system is being used toadd virtual advertisements to the video, the operator may need toidentify which locations in the stadium will be used for the virtualadvertisements and what advertisements to add. Some systems allow forthe inclusion or exclusions of certain colors. This information can alsobe set up during step 350. Other user defined variables such as graphiccolor, sizing and various parameters can be set up in step 350.

In step 352, the system is registered. Registration is the process ofdefining how to interpret data from a sensor (a camera being one type ofsensor, a camera view sensor being another type of sensor). The sensorsdescribed above output data, for example, related to position. Sinceposition is relative, the system needs a reference from which todetermine position or location. Thus, in order to be able to usepositional data, the system needs to know how to interpret thepositional data to make use of the information. For example, acoordinate system needs to be set up, the locations of each of thesensors need to be noted and the sensors need to be calibrated so thedata can be interpreted in light of the coordinate system. Theregistration process is different for each system and so the processneeds to be tailored to the individual sensors. Many of the referencesnoted above describe registration processes.

In step 354, the system is operated during the event. Step 354 caninclude the steps of FIG. 4, FIG. 6 and/or FIG. 7, as well as methodsfor operating other systems within the spirit of the present invention

FIGS. 6 and 7 are flow charts which explain the operation of anotherembodiment of the present invention. The method of FIGS. 6 and 7 can beused with the system of FIG. 1 or the system of FIGS. 2 and 3. In oneembodiment, the steps of FIG. 6 are performed at the event and the stepsof FIG. 7 are performed at the studio, remote from the event. In step402, sensors 108, 112 and 116 are used to acquire camera view data. Instep 404, the system receives a time code. Typically the time code isreceived from the broadcast production truck. In step 406, the time codeis associated with the appropriate camera view data. For example, dataconcentrator 122 will associate a time code received from time codegenerator 124 with the appropriate camera view data. Alternatively, thetime code is added to the program video by time code inserter 132, anddata inserter 140 inserts the appropriate camera view data in the samevertical blanking interval as the appropriate time code. In step 408,the system determines which camera is tallied. In step 410, the systemcombines the time code with the video. For example, in FIG. 2, time codeinserter 132 inserts the time code into the program video. The time codecan be inserted into the vertical blanking interval of the program videoor can be inserted into another portion of the program video. In step412, the system combines the camera view data with the video. In step414, the system combines the tally indication with the video. In step416, the system combines the registration information with the video.Looking at FIG. 2, data inserter 140 can be used to perform steps 412,414 and 416. In one embodiment, steps 412, 414 and 416 are performedsimultaneously. In step 418, the video, camera view data, time code,tally indication and registration information are transmitted over themain backhaul and/or back up backhaul as discussed above.

FIG. 7 describes the steps of one embodiment that are performed at thestudio. In step 502, the location of the target area to be enhanced isidentified. There are many alternatives for performing step 502. Forexample, sensors can be used to determine the three dimensional locationof an object that is moving at an event. Such systems of sensors areknown in the art and are described in U.S. patent application Ser. Nos.08/585,145, 08/786,108 and 09/041,238, all of which are cited above.Alternatively, U.S. patent application Ser. No. 09/160,534 describes asystem in which the operator manually types in an indication of a threedimensional location at the event. Other systems can also be used todetermine the location of the target area to be enhanced. In oneembodiment, step 502 is not performed as part of the present invention.

The steps of FIG. 6 and FIG. 7 are continuously performed throughout theevent. In one embodiment, the steps of FIG. 6 and FIG. 7 are performedon a field by field basis, a frame by frame basis or some other period.

In step 504, the studio receives the data and video transmitted in step418. In step 506, data is read, decoded or otherwise separated from thevideo, if necessary. For example, step 506 could include using receiver186 to access the data in the vertical blanking interval of a videosignal. Step 508 includes determining the position of the target area inthe video. Different systems for enhancing a video have differentmethods for determining the position of the target area in the video.Each of the systems cited above have various means for determining theposition in the video. Any of those means for determining the positioncan be used with respect to step 508. For example, many of the prior artsystems use a matrix to convert a three dimensional location to a twodimensional position in the video. In step 510, the video is enhanced.In one embodiment, a graphic is blended with the target area of thevideo. In another embodiment, a graphic can be blended with the video ata location adjacent to or logically near the target area determined instep 508.

In another embodiment, rather than determine the position within thevideo and enhance the video during the live event, the video and all thedata is recorded by recorder 178 (and, optionally recorder 184) using avideo tape, or other medium. The components of FIG. 3 can be used toperform the steps of FIG. 7 subsequent to the end of the event in orderto post process the video tape. By recording all of the camera viewdata, time codes and registration information, the system of FIG. 3 caneasily post process the data.

Additionally, because the equipment depicted in FIG. 3 is located at astudio remote from the event, immediately following the end of the eventthe equipment of FIG. 3 can be used for another event. The equipmentdoes not need to be transported before subsequent use and thebroadcaster need not purchase an expensive truck to transport theequipment. In one alternative, the processors depicted in FIG. 3 can belarge main frames or super computers so that more than one event can beprocessed at the same time. It is also contemplated that a studiocomplex can have multiple studios within the complex for simultaneouslyprocessing/broadcasting many events. Each studio can contain thecomponents of FIG. 3 so that multiple events can be processed andbroadcast at the same time.

In one embodiment, the systems described herein operate in real time toenhance live video. Live video can be delayed a small amount of time andstill be considered live. For example, a live television broadcast canbe delayed one third to one half of a second and still be consideredlive. Other small delays are also acceptable. However, the frame delayshould be constant. That is, if the video processing requires a fifteenframe delay, then each frame for broadcast should be delayed no morethan fifteen frames. The delay cannot grow as the event progresses.

In one embodiment, data transmitted from the site of the event to thestudio could include the three dimensional location of one or moretargets. Examples of targets include pucks, balls, players, cars,animals, field positions, lines or other markings on the field of play,locations of objects of interest, etc. This data can be sent to thestudio and used to enhance video and/or audio. That is, the dataconcentrator can send the appropriate sensor data to a processor at theevent. The processor can use the sensor data to determine the threedimensional location of a target. This three dimensional location can betransmitted to the studio and used by another computer to determine atwo dimensional pixel location of the target's position on a frame orfield of video.

The foregoing detailed description of the invention has been presentedfor purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise form disclosed, andobviously many modifications and variations are possible in light of theabove teaching. The described embodiments were chosen in order to bestexplain the principles of the invention and its practical application tothereby enable others skilled in the art to best utilize the inventionin various embodiments and with various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the claims appended hereto.

We claim:
 1. A method for enhancing a video presentation of a liveevent, comprising the steps of: receiving camera view data and firstvideo during said live event, said camera view data and said first videoare received at a location remote from said live event, said camera viewdata and said first video are received from a video production facilityfor said first video, said video production facility is located at saidlive event, said camera view data pertains to a first camera, said firstcamera captures said first video; determining a position in said firstvideo based on said received camera view data, said step of determininga position is performed at said location remote from said live event;and enhancing said first video based on said determined position in saidfirst video, said step of enhancing is performed at said location remotefrom said live event, said step of enhancing is performed during saidlive event.
 2. A method according to 1, further comprising the steps of:sensing said camera view data at said live event and during said liveevent; transmitting said camera view data, during said live event, tosaid video production facility; and transmitting said camera view data,during said live event, from said video production facility to saidlocation remote from said live event.
 3. A method according to claim 2,further comprising the steps of: sensing additional camera view data foradditional cameras at said live event; and transmitting said additionalcamera view data to said video production facility.
 4. A methodaccording to claim 2, further comprising the step of: associating one ormore time codes with said camera view data in order to synchronize saidcamera view data with said first video, said step of associating isperformed at said video production facility, said step of receivingincludes receiving said one or more time codes, said step of determininguses said time codes.
 5. A method according to 1, further comprising thestep of: receiving registration information multiple times from saidlive event, said registration information is received at said locationremote from said live event, said registration information pertains tosaid camera view data, said step of determining uses said registrationinformation to determine said position.
 6. A method according to claim1, wherein: said camera view data and said first video are received fromsaid video production facility via a backhaul.
 7. A method according to1, further comprising the step of: receiving information pertaining to athree dimensional position of an object at said live event, said step ofdetermining uses said information to determine said position.
 8. Amethod according to 1, further comprising the steps of: sensing saidcamera view data at said live event and during said live event;transmitting said camera view data, during said live event, to saidvideo production facility; transmitting said camera view data, duringsaid live event, from said video production facility to said locationremote from said live event; associating one or more time codes withsaid camera view data in order to synchronize said camera view data withsaid first video, said step of associating is performed at said videoproduction facility, said step of receiving includes receiving said oneor more time codes, said step of determining uses said time codes; andreceiving registration information multiple times from said live event,said registration information is received at said location remote fromsaid live event, said registration information pertains to said cameraview data, said step of determining uses said registration informationto determine said position.
 9. An apparatus for enhancing a videopresentation of a live event, comprising: a receiver at a locationremote from said live event, said receiver is in communication with avideo production facility at said live event in order to receive cameraview data and first video from said video production facility duringsaid live event, said camera view data pertains to a first camera, saidfirst camera captures said first video; and a video processing apparatusat said location remote from said live event, said video processingapparatus is in communication with said receiver, said video processingapparatus determines a position in said first video based on saidreceived camera view data, said video processing apparatus enhances saidfirst video based on said determined position in said first video duringsaid live event.
 10. An apparatus according 9, wherein: said receiverreceives registration information multiple times from said live event,said registration information pertains to said camera view data, saidvideo processing apparatus uses said registration information todetermine said position.
 11. An apparatus according 9, wherein: saidreceiver receives said camera view data and first video via a backhaul.12. An apparatus according to 9, wherein: said receiver receivesinformation pertaining to a three dimensional position of an object atsaid live event, said video processing apparatus uses said informationto determine said position.
 13. A method for enhancing videopresentations, comprising the steps of: receiving camera view data formultiple live events at a central location that is remote from said liveevents, said step of receiving is performed during said live events;determining positions in videos of said live events based on saidreceived camera view data, said step of determining positions isperformed at said central location; and enhancing said videos based onsaid determined positions, said step of enhancing is performed at saidcentral location, said step of enhancing is performed during said liveevents.
 14. A method according to claim 13, further comprising the stepsof: sensing said camera view data at said live events, said camera viewdata pertains to cameras at said live events that capture said videos ofsaid live events; and transmitting said camera view data to said centrallocation.
 15. A method according to claim 14, further comprising thesteps of: associating time codes with said camera view data in order tosynchronize said camera view data with said videos, said step ofassociating is performed at said live events, said step of receivingincludes receiving said time codes, said step of determining uses saidtime codes.
 16. A method according to claim 13, wherein: said cameraview data is received via backhauls from said live events.
 17. A methodaccording to claim 13, wherein: said camera view data is received fromvideo production facilities at said live events.
 18. A method accordingto 13, further comprising the step of: receiving registrationinformation multiple times from said live events, said registrationinformation is received at said central location, said registrationinformation pertains to said camera view data, said step of determininguses said registration information to determine said positions.
 19. Amethod according to 13, further comprising the step of: receivinginformation at said central location from said live events, saidinformation pertains to three dimensional locations of objects at saidlive events, said step of determining uses said information to determinesaid positions.
 20. An apparatus for enhancing video presentations oflive events, comprising: a set of one or more receivers in communicationwith transmitters at said live events in order to receive camera viewdata for said live events during said live events, said one or morereceivers are at a central location remote from said live events; andone or more video processing units at said central location incommunication with one or more of said receivers, said video processingunits determine positions in videos of said live events based on saidreceived camera view data, said video processing units enhance saidvideos based on said determined positions during said live event.
 21. Anapparatus according to claim 20, wherein: said receivers are incommunication with said transmitters via backhauls from said liveevents.
 22. An apparatus according to claim 20, wherein: saidtransmitters are located at video production facilities at said liveevents.
 23. An apparatus according to claim 20, wherein: said receiversreceive registration information multiple times from said live events,said registration information pertains to said camera view data, saidone or more video processing units use said registration information todetermine said positions.
 24. An apparatus according to claim 20,wherein: said receivers receive information from said live events thatpertains to three dimensional locations of objects at said live events,said one or more video processing units use said information todetermine said positions.
 25. A method for enhancing a videopresentation of a live event, comprising the steps of: receiving firstcamera view data during said live event from said live event, said firstcamera view data is received at a location remote from said live event,said first camera view data pertains to a first camera, said firstcamera captures first video of said live event; receiving registrationinformation multiple times from a first location at said live event,said registration information is received at said location remote fromsaid live event, said registration information pertains to said firstcamera view data; determining a position in said first video based onsaid received first camera view data and said registration information,said step of determining a position is performed at said location remotefrom said live event; and enhancing said first video based on saiddetermined position in said first video, said step of enhancing isperformed at said location remote from said live event, said step ofenhancing is performed during said live event.
 26. A method according toclaim 25, further comprising the steps of: sensing said first cameraview data at said live event and during said live event; transmittingsaid first camera view data to said location remote from said liveevent; and transmitting said registration information from said firstlocation to said location remote from said live event.
 27. A methodaccording to claim 26, further comprising the steps of: sensingadditional camera view data for additional cameras at said live event;transmitting said additional camera view data to a video productionfacility for said first video at said live event; transmitting saidfirst camera view data to said video production facility, said step oftransmitting said first camera view data to said location remote fromsaid live event includes transmitting said first camera view data fromsaid video production facility, said video production facility is atsaid first location.
 28. A method according to claim 27, furthercomprising the step of: associating one or more time codes with saidfirst camera view data in order to synchronize said first camera viewdata with said first video, said step of associating is performed atsaid first location, said step of receiving includes receiving said oneor more time codes, said step of determining uses said time codes.
 29. Amethod according to claim 28, wherein: said first camera view data andsaid first video are transmitted to said location remote from said liveevent via a backhaul.
 30. A method according to claim 25, wherein: saidstep of receiving registration information multiple times includesperiodically receiving said registration information.
 31. A method forenhancing a video presentation of a live event, comprising the steps of:sensing, during said live event, camera view data for multiple camerasat said live event, said multiple cameras capture video of said liveevent; transmitting, during said live event, said camera view data forsaid multiple cameras to a first location at said live event;transmitting, during said live event, at least a first subset of saidcamera view data from said first location to a second location, saidsecond location is remote from said live event; determining a positionin a first video based on said transmitted camera view data, said stepof determining a position is performed at said second location, saidfirst video is from one of said multiple cameras; and enhancing saidfirst video based on said position in said first video, said step ofenhancing is performed at said second location, said step of enhancingis performed during said live event.
 32. A method according to claim 31,wherein: said first location is a video production facility for saidfirst video.
 33. A method according to claim 31, wherein: said firstsubset of said camera view data is transmitted from said first locationto said second location via a backhaul.
 34. A method according to claim31, further comprising the step of: associating time codes with saidcamera view data in order to synchronize said camera view data with saidvideo from said multiple camera, said step of associating is performedat said first location.
 35. A method for enhancing a video presentationof a live event, comprising the steps of: receiving video for a firstcamera, said video is received at a location remote from said liveevent, said video is received from said live event via a backhaul, saidvideo depicts said live event, said video is received during said liveevent; receiving camera view data at said location remote from said liveevent during said live event, said camera view data pertains to saidfirst camera, said camera view data is transmitted with said video viasaid backhaul; determining a position in said video based on saidtransmitted camera view data, said step of determining a position isperformed at said location remote from said live event; and enhancingsaid video based on said determined position in said video, said step ofenhancing is performed at said location remote from said live event,said step of enhancing is performed during said live event.
 36. A methodaccording to 35, further comprising the steps of: sensing said cameraview data at said live event; and transmitting said camera view data tosaid location remote from said live event via said backhaul.
 37. Amethod according to claim 36, further comprising the step of:associating time codes with said camera view data in order tosynchronize said camera view data with said video, said step ofassociating is performed at a production facility for said video at saidlive event, said step of transmitting includes transmitting said cameraview data from one or more camera view sensors to said productionfacility prior to said step of associating and transmitting said cameraview data to said location remote from said live event via said backhaulafter said step of associating.